\documentclass[10pt,twocolumn,letterpaper]{article}

\usepackage{cvpr}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{dsfont}
\usepackage{bm}
\usepackage{bbm}
\usepackage{amsfonts,amscd}
\usepackage{amssymb,amsmath,amsthm,enumerate}
\usepackage{physics}

\usepackage{tikz}
\usetikzlibrary{calc}
\usetikzlibrary{shapes,arrows}

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

% Include other packages here, before hyperref.

% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex.  (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[breaklinks=true,bookmarks=false]{hyperref}

\cvprfinalcopy % *** Uncomment this line for the final submission

\def\cvprPaperID{****} % *** Enter the CVPR Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}

% Pages are numbered in submission mode, and unnumbered in camera-ready
%\ifcvprfinal\pagestyle{empty}\fi
\setcounter{page}{1}
\begin{document}

%%%%%%%%% TITLE
\title{Deep Reinforcement Learning for Bearing-only Radiolocation}

\author{Cedrick Argueta \\
{\tt\small cdrckrgt@stanford.edu}
% For a paper whose authors are all at the same institution,
% omit the following lines up until the closing ``}''.
% Additional authors and addresses can be added with ``\and'',
% just like the second author.
% To save space, use either the email address or home page, not both
}

\maketitle
%\thispagestyle{empty}

%%%%%%%%% ABSTRACT
\begin{abstract}
Unauthorized drones present a danger to airports and disaster areas.
Localization and tracking of these unauthorized drones reduces some of this danger.
It is possible to use a drone outfitted with with commericial antennas and radios to autonomously localize other drones.
In this work, we show preliminary results detailing how a drone with this equipment may use deep reinforcement learning to perform path planning in a localization task.
\end{abstract}

\section{Details for CS229 Project}
I am planning on making some improvements to a project that I'm working on for my honors thesis.
The other sections of this paper are part of an early draft of the paper I'm writing up, which represent the basis for the CS229 project I want to do.
Currently, I'm running experiments to finish this paper and submit it somewhere.
I don't plan on using this \textit{exact} project for CS229, but instead explore some possible improvements to this I could incorporate into another paper down the line.
I have a well maintained simulation package for drone seeking problems that I will be using for this project, found here \cite{PyFEBOL}.

The gist of this problem is as follows: a seeker drone is trying to localize a target drone.
The target drone is localized when the seeker drone has a proper idea of where the target drone is -- the way this is done in this work is with a particle filter.
When the particles are concentrated on the target drone's actual position, the target is localized.
The seeker makes measurements in a search environment and uses these measurements to make decisions about where to move.
These measurements are noisy relative bearing measurements of the target's position.
The goal is to find a policy for the seeker that localizes the target quickly and with high accuracy and precision.

The work this is based on used MCTS for planning \cite{dronehunter}, and my mentor had the idea to use reinforcement learning to improve performance.
Prior work suggests this is possible for fixed-wing drones \cite{kyle1} and drones in source-seeking applications \cite{learning_to_seek}.
I already have an implementation of DQN that can solve this problem, so I want to try different deep reinforcement learning algorithms.
I primarily want to try either a recurrent network-based algorithm like DRQN or a continuous control algorithm like DDPG.
The motivation behind DRQN would be that this problem is partially observed, and integrating information over time with an RNN might improve planning performance over DQN.
Increased training time with the RNN scares me, so maybe this is where other CS229 concepts could help -- some sort of dimenionality reduction or other preprocessing on the particle filter input would help keep training time realistic.
DDPG would perhaps not be a strict improvement on performance but would instead allow the seeker drone to take finer actions than the finite limit I've currently set.
Since it might take longer for a DDPG-based planner to converge to a good solution here, it might also make sense to use some sort of dimensionality reduction on the filter input.

The evaluation of the agent is based off that in \cite{dronehunter}.
The seeker should move to gather information from its sensors and concentrate particles in the filter to the true position of the target.
This can be quantified by minimization of entropy and tracking error.
In practice, maintaining distance from the target is useful -- you avoid detection from the target, and prevent in-air collisions.
Then a `good' agent minimizes these three things.
The goal is to show that for different collision rates, a learning-based planner will have a lower tracking error than an MCTS-based planner.



\section{Introduction}
The use of drones has become widespread in recent years.
In the civil sector, drones can be used for disaster relief or surveillance.
In the commercial sector, drones can be used for package delivery, photography or film production, or agriculture.
Drones are also often used in the military for reconnaissance and as weapons.

The popularity of drones brings with it several challenges.
Drones near airports pose a threat to aircraft operations, and carry the potential for a terrorist attack \cite{marketwatch_drone}.
This year to date, the UK Airprox Board has recorded 18 "Airprox" events in which drones may have compromised the safety of aircraft \cite{airprox}.
Currently, the FAA prohibits drones from operating near rescue efforts in natural disasters like hurricanes or wildfires \cite{faa}.
In these situations, it would be beneficial to localize the position of the offending drone.
Then it'd be possible to intercept and deal with the drone in a timely and safe manner.

Tracking a drone can be done by monitoring its radio telemetry.
Drones usually emit radio signals to their operators, like speed, position, battery life, and other vital data. 
A seeker drone outfitted with radio receivers and antennas can home in on these radio signals and find the target drone's position.
The sensor modality we consider is a directional antenna and an omnidirectional antenna, which has been shown to provide accurate bearing measurements on drones \cite{sensor_modality}.
This combination of sensors allows for accurate bearing estimates of the target signal.
While it is possible to track drones on the ground or with stationary trackers, a drone-based tracker would be as mobile as the target and be able to follow it for longer distances.

This work builds heavily off of \cite{dronehunter} and \cite{kyle2} in that it has the same goal -- improve drone tracking performance over greedy, one-step planners.
A greedy solution would execute the best myopic action at every time step, without planning ahead to the next.
The drone's path planning system aims to find a policy that chooses the best overall actions, perhaps sometimes making suboptimal myopic decisions in exchange for better hyperopic path planning.

This formulation of the problem has two drones: a seeker drone and a target drone. 
The seeker drone's objective is to track the moving target drone by capturing emissions by the target drone's radio.
Existing solutions using Monte Carlo tree search (MCTS) often require large amounts of memory and computational power that might be lacking on drone avionics boards.
Previous work  shows how neural networks can be used for guidance of drones to waypoints with much smaller memory footprints than traditional Markov decision process (MDP) solutions in obstacle avoidance and waypoint finding \cite{kyle2}.
This work applies the same methodology to the drone localization problem.
We show that it is possible to train a deep reinforcement learning (RL) algorithm, namely Deep Q-Network (DQN), to perform path planning in a bearing-only case.

\textit{Our main contribution is the development of a learning-based planner that achieves a lower tracking error and near-collision rate compared to other planners.}

\section{Related Work}
An MCTS solution to drone localization is presented by Dressel and Kochenderfer in \cite{dronehunter}.
The sensor modality used in this work consists of two directional antennas that are used to compare signal strength in front of or behind the drone.
Our work differs in that we use a directional antenna and an omnidirectional antenna to provide pseudo-bearing measurements to the controller.

Using neural networks for path planning in drones is explored in \cite{kyle2}.
In this work, the lookup table for the discrete value iteration solution is represented by a neural network.

Reinforcement learning for unmanned aerial vehicle (UAV) control is applied in \cite{kyle1}.
Here, a fixed-wing UAV used for wilfire monitoring is controlled using deep RL.

Reinforcement learning for controlling multirotor UAVs is also explored in \cite{learning_to_seek}.

\section{Mathematical Models}
\label{sec:models}
This section described the mathematical models used to formalize the system.

\subsection{Drone Dynamics}
The drone dynamics are exactly those described in \cite{dronehunter}.
The seeker drone state at time $t$ is $x_t = [x_t^n, x_t^e, x_t^h]^\intercal$, where $x_t^n$ and $x_t^e$ are the seeker's north and east coordinates and $x_t^h$ is the seeker's heading measured east of north.
The state described here does not contain velocity or altitude as simplifying assumptions.
The drone follows a first-order motion model, so the state after applying a control input $u_t$ for duration $\Delta t$ the new state is
\begin{equation}
x_{t + \Delta t} = x_t + u_t\Delta t
\end{equation}

The target drone state at time $t$ is $\theta_t = [\theta_t^n, \theta_t^e]^\intercal$, where $\theta_t^n$ and $\theta_t^e$ are the target's north and east coordinates.
The target drone is assumed to move with a constant velocity $\dot{\theta} = [\dot{\theta_t^n}, \dot{\theta_t^e}]^\intercal$.
The drone follows a first-order motion model, so the state after $\Delta t$ is
\begin{equation}
\theta_{t + \Delta t} = \theta_t + \dot{\theta_t}\Delta t
\label{target_dynamics}
\end{equation}

\subsection{Sensor Model}
The bearing from the seeker drone to the target drone is 
\begin{equation}
\beta_t = \arctan{\frac{\theta_t^e - x_t^e}{\theta_t^n - x_t^n}}
\end{equation}
when measured east of north.
Configured properly, the directional antenna and omnidirectional antenna can give estimates of the relative bearing of the target drone.

At time $t$, the seeker drone makes measurement $z_t \sim \mathcal{N}(\beta_t - x_t^h, \sigma^2)$, which is a bearing in the interval $[0^{\circ}, 360^{\circ}]$.
It is assumed that these measurements are normally distributed about the true relative bearing with some variance to account for sensor error.

\subsection{Particle Filter}
The seeker drone maintains a belief of the possible location of the target drone, which is modeled with a particle filter.
Belief at time $t$ is represented by a set of $N$ particles, each representing a hypothesis of the target drone's pose.
Updates are made to the particle filter at every timestep to improve the belief's accuracy.

The belief update consists of three steps. 
The first step is the prediction step, where each particle is propagated according to the dynamics described in equation \ref{target_dynamics}.
Noise is added to the dynamics to prevent particle deprivation, a situation that arises when all particles converge to a hypothesis that doesn't accurately represent the true state.
The second step is the weighting step, where each particle is assigned a weight according to how probable an observation $z_t$ is given the particle's position.
The third step is resampling, where particles are sampled according to these weights with replacement.
In this work, we use stratified resampling to aid in maintaining an accurate estimate of the target while ensuring resiliency to particle deprivation.

\subsection{Belief Markov Decision Process}

The planning algorithm uses the partially observable Markov decision process (POMDP) framework for analysis.

A POMDP comprises a state space $\mathcal{S}$, an action space $\mathcal{A}$, a reward function $R$, and a transition function $T$ defining the transition between states.
An agent is unable to observe the true state $s_t$ directly and instead makes an observation $\omega \in \Omega$ conditioned on the true state.

Solving a POMDP consists of finding a policy $\pi^*$ such that
\begin{equation}
\pi^* = \argmax_{\pi}{\mathbb{E}\left [ \sum_{t=0}^{\infty}\gamma^{t}R(s_t, \pi(\omega_t)) \right ]}
\label{optimal_policy}
\end{equation}
where $\gamma$ is a discount factor.

POMDPs have a significant disadvantage when formalizing localization tasks.
Rewards that depend on belief of the true state of the system are often difficult to represent in the POMDP framework \cite{dronehunter}.
For this reason, we instead convert the POMDP to a belief-Markov decision process, or belief-MDP.

Belief-MDPs are similar to MDPs where the system state is instead a \textit{belief} of the true system state.
We hereafter model the problem as an MDP where each state is a tuple of the fully observable part of the true state and the belief of the partially observable part of the true state.

\subsection{Formulation}

\subsubsection{States}
Each state is a tuple $s_t = (b_t, x_t)$ where $b_t$ is the seeker's belief and $x_t$ is the seeker's pose.
The seeker's belief is the particle filter mentioned in the previous section.

\subsubsection{Actions}
While the belief-MDP framework is general enough to work with a continuous action space, this work focuses on the simpler discrete action space.
The seeker drone is allowed to travel with a constant velocity in 24 directions equally spaced in a radial pattern.

\subsubsection{Reward Function}
Our reward function for radiolocation captures the desire to maintain an accurate and precise estimate of the target's location \textit{while also} maintaining an acceptable distance from the target.

A precise belief is one that has low uncertainty over the target's position.
Minimization of this uncertainty is equivalent to minimization of the entropy of the belief distribution.
Particles in the filter are first discretized into $M$ bins.
Entropy can then be defined as:
\begin{equation}
H(b_t) = -\sum_{i = 1}^M\tilde{b}_t[i]\log\tilde{b}_t[i]
\label{entropy_unnormalized}
\end{equation}
where $\tilde{b}_t$ is the proportion of particles in each bin.
We normalize this entropy to be between $0$ and $1$, arriving at
\begin{equation}
H_{n}(b_t) = 1 - \frac{H(b_t)}{\log{M}}
\label{entropy_normalized}
\end{equation}.

An accurate belief is one that has a low tracking error with respect to the true target's state.
Tracking error is
\begin{equation}
E(b_t, \theta_t) = \Vert \mathbb{E}{[ b_t ]} - \theta_t \Vert
\label{tracking_unnormalized}
\end{equation}
but we again normalize the error to be between $0$ and $1$.
We choose to divide by the maximum error in the search domain, which for a domain of length $l$ is $l\sqrt{2}$.
Our normalized tracking error is then
\begin{equation}
E_{n}(b_t, \theta_t) = 1 - \frac{E(b_t)}{l\sqrt{2}}
\label{tracking_normalized}
\end{equation}.

Near-collisions are penalized to encourage the seeker to keep a safe distance.
The penalty term contains only the belief of the belief of the target's position rather than the true target position.
This is to encourage the seeker to maintain a distance from the particles during evaluation.
If the belief is representative of the true state, then the seeker will maintain a safe distance.
If the belief is not representative of the true state, then the seeker will at least maintain a distance from the belief, which still might contain a noisy or partially accurate model of the target's motion.
Our near-collision penalty is
\begin{equation}
C(b_t, x_t) = 1 - \mathop{{}\mathbb{E}}_{b_t} \mathds{1} (\Vert x_t - \theta_t\Vert < d)
\label{collision_penalty}
\end{equation}
where $\mathds{1}$ is an indicator function and $d$ is the safe distance we wish the seeker to maintain.

The terms are combined to produce our full reward function:
\begin{equation}
R(b_t, x_t, \theta_t) = \lambda_1 H_{n}(b_t) + \lambda_2 E_{n}(b_t, \theta_t) + \lambda_3 C(b_t, x_t)
\label{reward_function}
\end{equation}
where $\lambda_1$, $\lambda_2,$ and $\lambda_3$ are coefficients controlling the importance of each term.

\section{Planning Algorithm}
We use deep reinforcement learning to approximate the optimal solution to this belief-MDP.
\subsection{Deep Q-Networks}
In the Deep Q-network (DQN) algorithm, state-action pairs are assigned a value $Q(s, a)$ representing the value of taking that action from that state \cite{dqn}.
This value is defined by the Bellman equation
\begin{equation}
Q(s, a) = R(s) + \gamma\sum_{s' \in \mathcal{S}}p(s' | s, a)\max_{a' \in \mathcal{A}}Q(s', a')
\label{bellman_equation}
\end{equation}
where $p(s' | s, a)$ is the probability of transitioning to state $s'$ from state $s$ after taking action $a$.
Because computing the true value of $Q$ for every state-action pair is intractable, we use a neural network to approximate $Q$.
We train this neural network by minimizing the Bellman error 
\begin{equation}
E_{Bellman} = R + \gamma\max_{a' \in \mathcal{A}}{Q(s', a'; \mathbf{w})} - Q(s, a; \mathbf{w})
\label{bellman_error}
\end{equation}
with gradient descent.
The optimal policy derived from the Bellman equation is then
\begin{equation}
\pi^*(s) \approx \argmax_{a \in \mathcal{A}}{Q(s, a; \mathbf{w})}
\label{bellman_optimal}
\end{equation}

In practice, we apply the double Q-learning modification from \cite{double_dqn} and dueling network architecture from \cite{dueling_dqn}.
\subsection{Network Architecture}
A neural network is used to approximate the $Q$ value function.
In a style similar to \cite{kyle1}, a two-stream architecture (Figure \ref{arch})is used.

The input to the neural network at time $t$ is the belief $b_t$ and the seeker's pose $x_t$.
The particle filter representing belief is first discretized to a 2d histogram.
Convolutional layers are then used to extract spatial information from this downsampled belief.
Only particle positions are used in this downsampling -- the mean of the velocity of all particles is concatenated to $x_t$.
The downsampling allows us to take advantage of the spatial relationships that particles have.

% Define block styles
\tikzstyle{input} = [text width=8em, text centered, minimum height=1em, minimum width=1cm]
\tikzstyle{block} = [rectangle, draw, 
    text width=8em, text centered, rounded corners, minimum height=1em, minimum width=1cm]
\tikzstyle{output} = [text width=8em, text centered, minimum height=1em, minimum width=1cm]
\tikzstyle{line} = [draw, -latex']
  
% % old diagram 
% \begin{tikzpicture}[node distance = .6cm, auto]
%     % Place nodes
%     \node [input] (input1) {\tiny $x_t^n$, $x_t^e$, $h_t$, $\hat{\dot{\theta_t^n}}$, $\hat{\dot{\theta_t^e}}$};
%     \node [block, below of=input1] (dense1) {\tiny Fully-Connected Layers};
%     \node [below of=dense1] (space1) {};
%     \node [below of=space1] (space2) {};
% 
%     \node [input, right of=input1, node distance=4cm] (input2) {\tiny Discretized Filter};
%     \node [block, below of=input2] (conv1) {\tiny Convolutional Layers };
%     \node [block, below of=conv1] (dense2) {\tiny Fully-Connected Layers};
% 
%     \node [block, below of=dense2] (dense3) at ($(conv1)!0.5!(space2)$) {\tiny Fully-Connected Layers};
%     \node [output, below of=dense3] (output) {\tiny $Q$-value Output};
%     % Draw edges
%     \path [line] (dense1) -- (dense3);
% 
%     \path [line] (conv1) -- (dense2);
%     \path [line] (dense2) -- (dense3);
% 
%     \path [line] (input1) -- (dense1);
%     \path [line] (input2) -- (conv1);
%     
%     \path [line] (dense3) -- (output);
% 
% \end{tikzpicture}

\begin{figure}
\centering 
\begin{tikzpicture}[node distance = .6cm, auto]
    % Place nodes
    \node [input] (input1) {\tiny $x_t^n$, $x_t^e$, $h_t$, $\hat{\dot{\theta_t^n}}$, $\hat{\dot{\theta_t^e}}$};
    \node [block, below of=input1] (dense1) {\tiny $200$ fc};
    \node [block, below of=dense1] (dense2) {\tiny $200$ fc};
    \node [block, below of=dense2] (dense3) {\tiny $100$ fc};
    \node [below of=dense3] (space1) {};
    \node [below of=space1] (space2) {};
    \node [below of=space2] (space3) {};

    \node [input, right of=input1, node distance=4cm] (input2) {\tiny Discretized Filter};
    \node [block, below of=input2] (conv1) {\tiny $32\ 2\times 2$ conv };
    \node [block, below of=conv1] (conv2) {\tiny $64\ 2\times 2$ conv };
    \node [block, below of=conv2] (conv3) {\tiny $128\ 2\times 2$ conv };
    \node [block, below of=conv3] (dense4) {\tiny $200$ fc};
    \node [block, below of=dense4] (dense5) {\tiny $200$ fc};
    \node [block, below of=dense5] (dense6) {\tiny $100$ fc};

    \node [below of=space3] (space4) {};
    \node [below of=dense6] (space5) {};

    \node [block, below of=dense6] (dense7) at ($(space3)!0.5!(dense6)$) {\tiny $200$ fc};
    \node [block, below of=space4] (dense8) {\tiny $200$ fc};
    \node [block, below of=space5] (dense9) {\tiny $200$ fc};
    \node [output, below of=dense9] (output) at ($(dense8)!0.5!(dense9)$) {\tiny $Q$-value Output};
    % Draw edges
    \path [line] (input1) -- (dense1);
    \path [line] (input2) -- (conv1);

    \path [line] (dense1) -- (dense2);
    \path [line] (dense2) -- (dense3);

    \path [line] (conv1) -- (conv2);
    \path [line] (conv2) -- (conv3);
    \path [line] (conv3) -- (dense4);
    \path [line] (dense4) -- (dense5);
    \path [line] (dense5) -- (dense6);

    \path [line] (dense3) -- (dense7);
    \path [line] (dense6) -- (dense7);

    \path [line] (dense7) -- (dense8);
    \path [line] (dense7) -- (dense9);
    
    \path [line] (dense8) -- (output);
    \path [line] (dense9) -- (output);

\end{tikzpicture}
\caption{A two-stream dueling architecture is used to approximate the $Q$ value function. \texttt{fc} denotes a fully-connected layer while \texttt{conv} denotes a convolutional layer.}
\label{arch}
\end{figure}




\section{Preliminary Results}

\subsection{Simulation Parameters}
Training the planner on the physical system is infeasible because of time constraints -- the training environment must be reset every episode, human intervention is required to replace batteries, and weight updates are limited by the frequency of actions.
Thus, the network is trained on a simulator that captures the essential aspects of the system described in section \ref{sec:models}.
The simulator code may be found at \cite{PyFEBOL}.

The seeker and target drones are modeled in a 200 $m \times$ 200 $m$ area, where the seeker begins each episode at the center of the area and the target begins at a randomly selected corner and travels to an adjacent corner at 1.7 $m/s$.
The seeker drone can move at 5 $m/s$ in 36 equally-spaced directions or take no action.
The particle filter has 2000 particles, uniformly distributed at initialization and pruned using systematic sampling.
The values of $\lambda_1$, $\lambda_2$, and $\lambda_3$ were found empirically.

\subsection{Baseline}
The planner is compared against two baselines found in \cite{dronehunter}, a UCT-based MCTS planner and a greedy planner.
Both these methods are online and require no training.
For this reason, the cost function for agents using these planners do not use the tracking error term, as this would involve incorporating information into the solution method that the planner does not have access to.

\subsection{Results}

\section{Conclusion}


%%%%%%%%% BODY TEXT
% \section{Introduction}
% 
% Please follow the steps outlined below when submitting your manuscript to
% the IEEE Computer Society Press.  This style guide now has several
% important modifications (for example, you are no longer warned against the
% use of sticky tape to attach your artwork to the paper), so all authors
% should read this new version.
% 
% %-------------------------------------------------------------------------
% \subsection{Language}
% 
% All manuscripts must be in English.
% 
% \subsection{Dual submission}
% 
% Please refer to the author guidelines on the CVPR 2019 web page for a
% discussion of the policy on dual submissions.
% 
% \subsection{Paper length}
% Papers, excluding the references section,
% must be no longer than eight pages in length. The references section
% will not be included in the page count, and there is no limit on the
% length of the references section. For example, a paper of eight pages
% with two pages of references would have a total length of 10 pages.
% {\bf There will be no extra page charges for CVPR 2019.}
% 
% Overlength papers will simply not be reviewed.  This includes papers
% where the margins and formatting are deemed to have been significantly
% altered from those laid down by this style guide.  Note that this
% \LaTeX\ guide already sets figure captions and references in a smaller font.
% The reason such papers will not be reviewed is that there is no provision for
% supervised revisions of manuscripts.  The reviewing process cannot determine
% the suitability of the paper for presentation in eight pages if it is
% reviewed in eleven.  
% 
% %-------------------------------------------------------------------------
% \subsection{The ruler}
% The \LaTeX\ style defines a printed ruler which should be present in the
% version submitted for review.  The ruler is provided in order that
% reviewers may comment on particular lines in the paper without
% circumlocution.  If you are preparing a document using a non-\LaTeX\
% document preparation system, please arrange for an equivalent ruler to
% appear on the final output pages.  The presence or absence of the ruler
% should not change the appearance of any other content on the page.  The
% camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
% the \verb'\cvprfinalcopy' command in the document preamble.)  Reviewers:
% note that the ruler measurements do not align well with lines in the paper
% --- this turns out to be very difficult to do well when the paper contains
% many figures and equations, and, when done, looks ugly.  Just use fractional
% references (e.g.\ this line is $095.5$), although in most cases one would
% expect that the approximate location will be adequate.
% 
% \subsection{Mathematics}
% 
% Please number all of your sections and displayed equations.  It is
% important for readers to be able to refer to any particular equation.  Just
% because you didn't refer to it in the text doesn't mean some future reader
% might not need to refer to it.  It is cumbersome to have to use
% circumlocutions like ``the equation second from the top of page 3 column
% 1''.  (Note that the ruler will not be present in the final copy, so is not
% an alternative to equation numbers).  All authors will benefit from reading
% Mermin's description of how to write mathematics:
% \url{http://www.pamitc.org/documents/mermin.pdf}.
% 
% 
% \subsection{Blind review}
% 
% Many authors misunderstand the concept of anonymizing for blind
% review.  Blind review does not mean that one must remove
% citations to one's own work---in fact it is often impossible to
% review a paper unless the previous citations are known and
% available.
% 
% Blind review means that you do not use the words ``my'' or ``our''
% when citing previous work.  That is all.  (But see below for
% techreports.)
% 
% Saying ``this builds on the work of Lucy Smith [1]'' does not say
% that you are Lucy Smith; it says that you are building on her
% work.  If you are Smith and Jones, do not say ``as we show in
% [7]'', say ``as Smith and Jones show in [7]'' and at the end of the
% paper, include reference 7 as you would any other cited work.
% 
% An example of a bad paper just asking to be rejected:
% \begin{quote}
% \begin{center}
%     An analysis of the frobnicatable foo filter.
% \end{center}
% 
%    In this paper we present a performance analysis of our
%    previous paper [1], and show it to be inferior to all
%    previously known methods.  Why the previous paper was
%    accepted without this analysis is beyond me.
% 
%    [1] Removed for blind review
% \end{quote}
% 
% 
% An example of an acceptable paper:
% 
% \begin{quote}
% \begin{center}
%      An analysis of the frobnicatable foo filter.
% \end{center}
% 
%    In this paper we present a performance analysis of the
%    paper of Smith \etal [1], and show it to be inferior to
%    all previously known methods.  Why the previous paper
%    was accepted without this analysis is beyond me.
% 
%    [1] Smith, L and Jones, C. ``The frobnicatable foo
%    filter, a fundamental contribution to human knowledge''.
%    Nature 381(12), 1-213.
% \end{quote}
% 
% If you are making a submission to another conference at the same time,
% which covers similar or overlapping material, you may need to refer to that
% submission in order to explain the differences, just as you would if you
% had previously published related work.  In such cases, include the
% anonymized parallel submission~\cite{Authors14} as additional material and
% cite it as
% \begin{quote}
% [1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
% Supplied as additional material {\tt fg324.pdf}.
% \end{quote}
% 
% Finally, you may feel you need to tell the reader that more details can be
% found elsewhere, and refer them to a technical report.  For conference
% submissions, the paper must stand on its own, and not {\em require} the
% reviewer to go to a techreport for further details.  Thus, you may say in
% the body of the paper ``further details may be found
% in~\cite{Authors14b}''.  Then submit the techreport as additional material.
% Again, you may not assume the reviewers will read this material.
% 
% Sometimes your paper is about a problem which you tested using a tool which
% is widely known to be restricted to a single institution.  For example,
% let's say it's 1969, you have solved a key problem on the Apollo lander,
% and you believe that the CVPR70 audience would like to hear about your
% solution.  The work is a development of your celebrated 1968 paper entitled
% ``Zero-g frobnication: How being the only people in the world with access to
% the Apollo lander source code makes us a wow at parties'', by Zeus \etal.
% 
% You can handle this paper like any other.  Don't write ``We show how to
% improve our previous work [Anonymous, 1968].  This time we tested the
% algorithm on a lunar lander [name of lander removed for blind review]''.
% That would be silly, and would immediately identify the authors. Instead
% write the following:
% \begin{quotation}
% \noindent
%    We describe a system for zero-g frobnication.  This
%    system is new because it handles the following cases:
%    A, B.  Previous systems [Zeus et al. 1968] didn't
%    handle case B properly.  Ours handles it by including
%    a foo term in the bar integral.
% 
%    ...
% 
%    The proposed system was integrated with the Apollo
%    lunar lander, and went all the way to the moon, don't
%    you know.  It displayed the following behaviours
%    which show how well we solved cases A and B: ...
% \end{quotation}
% As you can see, the above text follows standard scientific convention,
% reads better than the first version, and does not explicitly name you as
% the authors.  A reviewer might think it likely that the new paper was
% written by Zeus \etal, but cannot make any decision based on that guess.
% He or she would have to be sure that no other authors could have been
% contracted to solve problem B.
% \medskip
% 
% \noindent
% FAQ\medskip\\
% {\bf Q:} Are acknowledgements OK?\\
% {\bf A:} No.  Leave them for the final copy.\medskip\\
% {\bf Q:} How do I cite my results reported in open challenges?
% {\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\
% 
% 
% 
% \begin{figure}[t]
% \begin{center}
% \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
%    %\includegraphics[width=0.8\linewidth]{egfigure.eps}
% \end{center}
%    \caption{Example of caption.  It is set in Roman so that mathematics
%    (always set in Roman: $B \sin A = A \sin B$) may be included without an
%    ugly clash.}
% \label{fig:long}
% \label{fig:onecol}
% \end{figure}
% 
% \subsection{Miscellaneous}
% 
% \noindent
% Compare the following:\\
% \begin{tabular}{ll}
%  \verb'$conf_a$' &  $conf_a$ \\
%  \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
% \end{tabular}\\
% See The \TeX book, p165.
% 
% The space after \eg, meaning ``for example'', should not be a
% sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
% \verb'\eg' macro takes care of this.
% 
% When citing a multi-author paper, you may save space by using ``et alia'',
% shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
% However, use it only when there are three or more authors.  Thus, the
% following is correct: ``
%    Frobnication has been trendy lately.
%    It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
%    Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''
% 
% This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
% because reference~\cite{Alpher03} has just two authors.  If you use the
% \verb'\etal' macro provided, then you need not worry about double periods
% when used at the end of a sentence as in Alpher \etal.
% 
% For this citation style, keep multiple citations in numerical (not
% chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
% \cite{Alpher02,Alpher03,Authors14}.
% 
% 
% \begin{figure*}
% \begin{center}
% \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
% \end{center}
%    \caption{Example of a short caption, which should be centered.}
% \label{fig:short}
% \end{figure*}
% 
% %------------------------------------------------------------------------
% \section{Formatting your paper}
% 
% All text must be in a two-column format. The total allowable width of the
% text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
% cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
% $\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
% first page) should begin 1.0 inch (2.54 cm) from the top edge of the
% page. The second and following pages should begin 1.0 inch (2.54 cm) from
% the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
% cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
% paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
% page.
% 
% %-------------------------------------------------------------------------
% \subsection{Margins and page numbering}
% 
% All printed material, including text, illustrations, and charts, must be kept
% within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
% high.
% Page numbers should be in footer with page numbers, centered and .75
% inches from the bottom of the page and make it start at the correct page
% number rather than the 4321 in the example.  To do this fine the line (around
% line 23)
% \begin{verbatim}
% %\ifcvprfinal\pagestyle{empty}\fi
% \setcounter{page}{4321}
% \end{verbatim}
% where the number 4321 is your assigned starting page.
% 
% Make sure the first page is numbered by commenting out the first page being
% empty on line 46
% \begin{verbatim}
% %\thispagestyle{empty}
% \end{verbatim}
% 
% 
% %-------------------------------------------------------------------------
% \subsection{Type-style and fonts}
% 
% Wherever Times is specified, Times Roman may also be used. If neither is
% available on your word processor, please use the font closest in
% appearance to Times to which you have access.
% 
% MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
% the first page. The title should be in Times 14-point, boldface type.
% Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
% adverbs; do not capitalize articles, coordinate conjunctions, or
% prepositions (unless the title begins with such a word). Leave two blank
% lines after the title.
% 
% AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
% and printed in Times 12-point, non-boldface type. This information is to
% be followed by two blank lines.
% 
% The ABSTRACT and MAIN TEXT are to be in a two-column format.
% 
% MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
% double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
% inch or 0.422 cm). Make sure your text is fully justified---that is,
% flush left and flush right. Please do not place any additional blank
% lines between paragraphs.
% 
% Figure and table captions should be 9-point Roman type as in
% Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.
% 
% \noindent Callouts should be 9-point Helvetica, non-boldface type.
% Initially capitalize only the first word of section titles and first-,
% second-, and third-order headings.
% 
% FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
% should be Times 12-point boldface, initially capitalized, flush left,
% with one blank line before, and one blank line after.
% 
% SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
% should be Times 11-point boldface, initially capitalized, flush left,
% with one blank line before, and one after. If you require a third-order
% heading (we discourage it), use 10-point Times, boldface, initially
% capitalized, flush left, preceded by one blank line, followed by a period
% and your text on the same line.
% 
% %-------------------------------------------------------------------------
% \subsection{Footnotes}
% 
% Please use footnotes\footnote {This is what a footnote looks like.  It
% often distracts the reader from the main flow of the argument.} sparingly.
% Indeed, try to avoid footnotes altogether and include necessary peripheral
% observations in
% the text (within parentheses, if you prefer, as in this sentence).  If you
% wish to use a footnote, place it at the bottom of the column on the page on
% which it is referenced. Use Times 8-point type, single-spaced.
% 
% 
% %-------------------------------------------------------------------------
% \subsection{References}
% 
% List and number all bibliographical references in 9-point Times,
% single-spaced, at the end of your paper. When referenced in the text,
% enclose the citation number in square brackets, for
% example~\cite{Authors14}.  Where appropriate, include the name(s) of
% editors of referenced books.
% 
% \begin{table}
% \begin{center}
% \begin{tabular}{|l|c|}
% \hline
% Method & Frobnability \\
% \hline\hline
% Theirs & Frumpy \\
% Yours & Frobbly \\
% Ours & Makes one's heart Frob\\
% \hline
% \end{tabular}
% \end{center}
% \caption{Results.   Ours is better.}
% \end{table}
% 
% %-------------------------------------------------------------------------
% \subsection{Illustrations, graphs, and photographs}
% 
% All graphics should be centered.  Please ensure that any point you wish to
% make is resolvable in a printed copy of the paper.  Resize fonts in figures
% to match the font in the body text, and choose line widths which render
% effectively in print.  Many readers (and reviewers), even of an electronic
% copy, will choose to print your paper in order to read it.  You cannot
% insist that they do otherwise, and therefore must not assume that they can
% zoom in to see tiny details on a graphic.
% 
% When placing figures in \LaTeX, it's almost always best to use
% \verb+\includegraphics+, and to specify the  figure width as a multiple of
% the line width as in the example below
% {\small\begin{verbatim}
%    \usepackage[dvips]{graphicx} ...
%    \includegraphics[width=0.8\linewidth]
%                    {myfile.eps}
% \end{verbatim}
% }
% 
% 
% %-------------------------------------------------------------------------
% \subsection{Color}
% 
% Please refer to the author guidelines on the CVPR 2019 web page for a discussion
% of the use of color in your document.
% 
% %------------------------------------------------------------------------
% \section{Final copy}
% 
% You must include your signed IEEE copyright release form when you submit
% your finished paper. We MUST have this form before your paper can be
% published in the proceedings.
% 
% Please direct any questions to the production editor in charge of these
% proceedings at the IEEE Computer Society Press: Phone (714) 821-8380, or
% Fax (714) 761-1784.

{\small
\bibliographystyle{unsrt}
\bibliography{ref}
}

\end{document}
